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Abstract 

x  A  set  of  principles  are  derived  from  the  ACT  theory  for  designing  intelligent  tutors:  identify  the 
goal  structure  of  the  problem  space,  provide  instruction  on  the  problem  solving  context,  provide 
immediate  feedback  in  errors,  minimize  working  memory  loads,  use  production  system  models  of  tho 
student,  adjust  the  grain  size  of  instruction  according  to  learning  principles,  enable  the  student  to 
approach  the  target  skills  by  successive  approximation,  and  promote  use  of  general  problem-solving 
rules  over  analogy.  These  principles  have  successfully  guided  our  design  of  tutors  for,  LISP  and 
geometry.  ' 
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There  has  been  and  continues  to  be  a  great  deal  of  hope  for  the  role  of  computers  in  education 
(Cohen,  1982;  Papert,  1980;  Taylor,  1980).  The  actual  record  of  accomplishment  is  still  quite  modest. 
Most  computer  education  takes  the  form  of  drill  and  practice  and  is  often  not  as  effective  as 
classroom  drill  and  practice.  There  has  always  been  the  hope  that  artificial  intelligence  techniques 
would  illuminate  computer  instruction.  The  buzz  word  of  a  decade  ago  was  "intelligent  computer 
assisted  instruction"  or  ICAI  (Carbonell,  1970).  The  relative  lack  of  progress  in  that  field  led  to  a 
disenchantment  with  that  term  and  we  now  see  new  words.  The  title  that  many  now  prefer,  inc'uding 
ourselves,  is  intelligent  tutoring  (Sleeman  and  Brown,  1982). 

The  basic  observation  that  motivates  the  intelligent  tutoring  approach  is  the  great  effectiveness 
of  private  human  tutors  over  either  classroom  instruction  or  standard  computer  education.  In 
comparisons  of  the  human  tutor  with  classroom  instruction  we  have  come  up  with  estimates  of  well 
jver  100%  improvement  in  effectiveness  (more  will  be  said  on  this  topic  at  the  end  of  the  paper).  The 
hope  of  intelligent  tutoring  is  to  find  some  way  of  "bottling"  the  skill  of  the  human  tutor  and  putting  it 
in  a  computer  tutor. 

Probably  the  expert  systems  methodology  would  be  the  most  straight  forward  way  of  applying 
artificial  intelligence  tc  intelligent  tutoring.  Treat  the  human  tutor  as  the  expert  whose  knowledge  has 
to  oe  extracted  and  build  an  expert  system  to  apply  that  methodology.  Work  such  as  that  of  Stevens, 
Collins,  and  Goldin  (1979)  on  teaching  topics  such  as  rainfall  seems  to  have  this  character.  However, 
human  tutoring  does  not  have  the  characteristics  of  a  domain  that  proves  susceptible  to  the  expert 
system  methodology,  it  is  not  a  relatively  circumscribed  knowledge  domain,  it  is  heavily  weighted  on 
natural  language  understanding,  and  human  tutors  show  enormous  differences  in  their  tutoring  style. 
Thus,  it  seems  unlikely  that  there  is  a  crystalized  expertise  to  be  captured  and  emulated. 

It  is  probably  for  this  reason  that  most  approaches  to  intelligent  tutoring  have  not  tried  to  really 
emulate  actual  tutors  (see  the  papers  in  Sleeman  and  Brown,  1982).  Rather  they  have  tried  to  identify 
principles  of  effective  tutoring  and  design  tutors  that  embody  these  principles.  These  tutors  have 
often  involved  expert  systems  as  submodules  in  the  tutor.  For  instance,  Brown,  Burton,  and  DeKleer 
(1982)  have  an  expert  circuit  analysis  system  as  part  of  their  SOPHIE  tutor  for  troubleshooting 
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circuits.  Such  a  system  can  reason  correctly  about  the  domain  and  the  correct  answer  is,  of  course, 
an  important  piece  of  information  in  instruction. 

We  feel  that  the  actual  pedagogical  design  of  the  tutor  has  to  be  based  on  detailed  cognitive 
models  of  how  students  solve  problems  and  learn.  Past  tutors  have  been  based  on  intuitions  about 
these  matters  but  not  on  cognitive  theories.  The  state  of  art  in  cognitive  science  has  reached  the 
point  where  we  now  are  able  to  produce  theories  capable  of  applications  to  intelligent  tutoring.  In  this 
paper  we  would  like  to  develop  a  set  of  principles  for  computer  tutoring  based  on  the  ACT  theory  of 
cognition  (Anderson,  198?).  This  theory  has  been  developed  at  a  sufficient  level  of  detail  that  it  is 
possible  to  produce  simulations  of  the  theory  that  actually  solve  problems  at  the  level  students  solve 
problems  and  which  learn  from  problem-solving  much  as  students  learn.  At  the  core  of  the  tutors  we 
have  developed  are  what  we  cal!  ideal  models  which  model  how  successful  students  solve  problems 
in  the  domain.  Such  models  are  one  step  more  constrained  than  the  expert  systems  of  most  tutors. 
Not  only  must  they  be  able  to  soive  the  problems  in  the  domain,  they  must  be  able  to  solve  them  the 
way  students  do. 

The  major  portion  of  this  paper  will  be  devoted  to  identifying  and  justifying  some  principles  for 
doing  intelligent  tutoring.  Then  the  last  two  sections  of  the  paper  will  describe  two  tutors  we  have 
built  partially  embodying  these  principles.  One  is  a  tutor  for  LISP  and  the  other  is  a  tutor  for  high 
school  geometry.  However,  before  embarking  on  any  of  this  we  should  state  a  major  limitation  on  the 
principles  that  we  will  articulate.  We  can  only  defend  their  application  to  a  relatively  restricted  set  of 
topics- -those  for  which  we  can  develop  idea*  student  models.  These  are  the  domains  of  high  school 
and  early  college  math  and  introductory  programming.  These  domains  are  relatively  sparse  in  their 
importation  of  extra-domain  knowledge  which  is  why  they  are  relatively  easy  to  develop  ideal  models 
for.  We  cannot  tutor  topics  for  which  we  lack  precise  student  models  in  the  same  way  as  we  can  tutor 
the  domains  for  which  we  do.  If  one  does  not  know  exactly  what  it  is  the  student  is  supposed  to  do,  it 
forces  one  to  back  off  into  a  different  tutoring  strategy. 


Throughout  this  paper  we  will  be  making  reference  to  observations  we  have  made  of  high 
school  students  learning  geometry  and  college  students  learning  to  program  in  LISP.  We  have 
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observed  four  students  spend  approximately  30  hours  studying  beginning  geometry  and  three 
students  similarly  spending  30  hours  learning  LISP.  These  sessions  have  been  ail  recorded  and  have 
been  subjected  to  varying  degrees  of  analysis.  Some  of  these  analyses  have  been  reported  in  a  series 
of  prior  publications  (Anderson,  1981;  Anderson,  1982;  Anderson,  1983a;  Anderson,  Farrell,  and 
Sauers,  1984;  Anderson.  Pirolli,  and  Farrell,  in  press).  This  data  base  has  served  as  a  rich  source  of 
information  about  the  acquisition  of  problem  solving  skill  and  has  heavily  influenced  our  design  of 
computer  tutors. 

Principle  1 :  Identify  the  Goal  Structure  of  the  Problem  Space 

According  to  the  ACT  theory,  and  indeed  most  cognitive  theories  of  problem  solving,  the 
problem  solving  behavior  is  organized  around  a  hierarchical  representation  of  the  current  goals.  It  is 
important  that  this  goal  structure  be  communicated  to  the  student  and  instruction  be  cast  in  terms  of 
it.  Below  we  discuss  the  fact  that  it  is  not  communicateo  in  typical  instruction  for  LISP  or  geometiy 
and  some  of  the  unfortunate  consequences  of  this  fact. 

Geometry 

Figure  1  shows  the  two-column  proof  form  that  is  almost  universally  used  in  geometry.  It  is 
basically  a  linear  structure  of  pairs  where  each  pair  is  a  statement  and  justification.  Typical 
instruction  encourages  the  belief  that  the  goal  structure  of  the  student  should  mimic  this  linear 
structure  -  that  at  any  point  in  the  proof  tne  student  will  have  generated  an  initial  part  of  the  structure 
and  the  current  goal  is  to  generate  the  next  line  of  the  structure. 

Insert  Figure  1  about  here 

There  are  two  serious  flaws  with  using  linear  proofs  as  goal  structures.  First  this  practice 
denies  the  validity  of  problem-solving  search.  It  encourages  the  idea  that  the  correct  next  line  should 
be  obvious,  but  finding  the  next  line  often  involves  considerable  planning  and  search.  Students 
engage  in  search  but  feel  bad  about  themselves  because  they  do.  Second,  search  in  such  a  linear 
structure  like  Figure  1  is  doomed  to  be  hopelessly  unguided.  If  the  only  constraint  is  to  generate  a 
legal  line,  the  search  space  for  the  correct  proof  is  hopelessly  large. 
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We  have  observed  students  fail  at  solving  geometry  problems  because  they  try  to  work  within 
this  linear  goal  structure.  In  our  limited  data  base  (four  students)  we  have  found  all  students 
eventually  overcome  lhis  linear  conception  and  arrive  at  a  better  conception  of  the  problem. 
However,  our  students  had  to  go  through  an  unnecessary  amount  of  frustration  before  developing  the 
necessary  insights  about  the  problem  space. 

Figure  2  illustrates  the  mental  representation  that  we  believe  a  successful  student  creates  for 
the  proof  of  a  geometry  problem.  The  conclusion  to  be  proven  is  related  to  the  givens  of  the  problem 
by  a  hierarchical  structure.  In  that  structure  rules  of  geometry  relate  givens  to  intermediate 
statements  and  these  statements  to  the  conclusion.  Hopefully,  the  readers  will  find  nothing  surprising 
in  this  representation  of  the  logical  structure  of  the  proof,  but  it  needs  to  be  emphasized  that 
conventional  instruction  does  not  communicate  this  structure  and  students  hardly  find  it  obvious. 
This  deficit  is  particula-ly  grievous  because  we  believe  that  the  successful  student’s  goal  structure  is 
much  more  closely  related  to  this  hierarchical  proof  structure  than  it  is  to  the  linear  structure  of  a 
two-column  proof. 

Insert  Figure  2  about  here 

Figure  3  illustrates  the  proof  in  Figure  2  embedded  in  a  set  of  additional  inferences  generated 
by  one  of  the  authors  (JRA)  in  constructing  the  proof.  The  inferences  are  numbered  in  the  order  that 
they  were  made.  The  structure  includes  many  inferences  that  were  made  in  an  attempt  to  construct 
the  proof  but  were  not  part  of  the  final  proof.  These  extra  inferences  reflect  some  of  the  search  that 
was  involved  in  producing  the  proof.  Because  standard  pedagogy  fails  to  communicate  either  the 
proof  structure  or  the  search  based  upon  it,  there  is  no  way  to  provide  instruction  about  this  critical 
aspect  of  the  learning  process.  This  aspect  is  entirely  a  matter  for  the.  student  to  induce.  Given  its 
complexity,  it  is  no  wonder  that  students  have  the  difficulties  they  do  with  geometry  proofs. 

Insert  Figure  3  about  here 
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LISP  Programming 

As  in  geometry,  the  goal  structure  underlying  writing  a  LISP  function  does  not  correspond  to 
the  syntax  of  the  problem  solution  and  yet  instruction  is  unfortunately  cast  in  terms  of  the  syntax.  Our 
studies  indicate  that  generating  a  LISP  program  is  largely  a  top-down,  planning  process  (Anderson, 
Farrell,  and  Sauers,  1982;  Anderson,  Pirolli,  and  Farrell,  in  press).  The  surface  form  of  most 
programming  languages  involves  a  linear  sequence  of  symbols.  Thus,  there  is  the  natural  danger  of 
casting  instruction  in  the  terms  of  this  linear  structure.  Fortunately,  more  enlightened  instruction  does 
emphasize  a  hierarchical,  structured  program.  There  is  evidence  that  this  is  a  better  instructional 
mode  (Shneiderman,  1980)  although  it  is  notoriously  difficult  to  prove  obviously  correct  hypotheses  in 
this  field  with  conventional  experimental  methodology  (Sheil,  1981). 

While  structured  programming  is  definitely  a  step  in  the  right  direction,  it  only  ameliorates  the 
basic  problem.  The  structured  program  itself  is  only  a  syntactic  object  which  will  have  an  imperfect 
correspondence  to  the  structure  of  the  programmer's  plan.  Consider  an  example  we  have  studied  at 
length  (Anderson,  Pirolli,  &  Farrell,  in  press)  from  learning  to  program  recursive  functions,  a  recursive 
function  that  calculates  the  intersection  of  two  lists; 

(DEFUN  INTERSECTION  (LIST1  LIST2) 

(COND  ((NULL  LIST1)  NIL) 

((MEMBER  (CAR  LIST1)  LIST2) 

(CONS  (CAR  LIST1 ) 

( INTERSECTION  (CDR  LIST1)  LIST2)) ) 

(T  (INTERSECTION  (CDR  LIST1)  LIST2 ) ) ) ) 

From  a  syntactic  point  of  view,  this  function  consists  of  a  conditional  structure  that  is  composed  of 
three  clauses,  each  consisting  of  a  condition  and  an  action.  The  student  is  encouraged  to  believe 
that  it  is  just  a  matter  of  "programmer’s  intuition"  that  leads  to  the  division  of  the  conditional  into  the 
right  set  of  three  conditions  and  actions. 

However,  there  are  very  precise  principles  that  underly  the  division  of  this  code  into  its 
components  (Soloway, 1980;  Rich  &  Shrobe,  1978).  This  is  an  example  of  what  we  call  the  CDR- 
recursion  plan.  This  plan  applies  when  an  argument  of  the  function  is  a  list  such  as  LIST  1 .  That  plan 
involves  coding  a  terminating  case  for  when  the  list  is  NIL  and  a  recursive  case  that  relates  the  result 


V  ' 

.  i 


.  t; , 


L 


of  the  function  applied  to  the  tail  (CDR)  of  the  list  to  the  result  of  the  function  applied  to  the  full  list.  In 
the  code  above,  the  first  clause  of  the  conditional  implements  the  terminating  case  and  the  second 
and  third  clauses  perform  the  recursive  step.  So,  the  plan  that  generated  the  conditional  consists  of 
two  major  subgoals,  not  three.  However,  it  is  necessary  to  break  the  recursive  step  into  two 
subcases -one  to  deal  with  the  situation  where  the  first  element  is  in  the  list  and  one  where  it  is  not. 
This  division  is  dictated  by  a  standard  list  search  plan. 

Experts  code  the  function  by  implementing  their  CDR-recursion  plan.  However,  this  powerful 
programming  technique  is  seldom  directly  taught.  Students  ere  given  explanations  of  how  the 
recursive  calls  work,  but  they  are  not  told  the  programming  plan  that  would  cause  someone  to  write 
these  calls  in  the  first  place.  We  have  seen  students  work  for  10  hours  with  some  popular  LISP  texts 
on  problems  like  INTERSECTION  and  not  figure  out  the  CDR-recursion  plan.  As  a  consequence,  they 
were  still  floundering  after  the  10  hours. 

Another  feature  of  problem-solving  in  LISP  is  that  the  programmer  often  has  to  interrupt  the 
coding  and  go  to  a  different  problem  space  for  algorithm  design  (Kant  &  Newell,  1982).  As  an 
example  of  algorithm  design  consider  how  the  student  determines  the  recursive  step  in  CDR- 
recursion.  A  typical  strategy  is  to  list  some  examples  of  what  the  value  of  the  function  is  for  a  set  cf 
example  lists  and  for  their  tails  and  try  to  induce  a  general  characterization  of  the  relationship 
between  the  value  o*  the  function  for  the  whole  list  and  for  the  tail  of  the  list.  Inducing  this 
characterization  frequently  involves  pattern-matching.  Listing  examples  and  pattern-matching  are 
not  code  generation  activities  but  are  critical  to  determining  the  code.  Again  standard  instruction  is 
silent  about  this  algorithm  design  process. 

Summary 

Traditional  instruction  does  not  explain  the  goal  structure  of  the  problem  solving  plan  nor  the 
search  involved  in  the  problem  solving.  Private  human  tutors  often  communicate  this  information  in 
their  real-time  comments  about  the  student’s  problem-solving.  For  instance,  all  three  of  the  LISP 
tutors  we  observed  suggested  enumerating  examples  to  discover  the  recursive  relationship.  A 
computer  tutor  should  strive  to  communicate  the  goal  structure  to  the  student  as  this  is  absolutely 


critical  to  effective  problem  solving. 

Principle  2:  Provide  Instruction  in  the  Problem-Solving  Context 

Students  appear  to  learn  information  better  if  they  are  presented  with  that  information  during 
problem  solving  rather  than  during  instruction  apart  from  the  the  problem  solving  context.  There  are 
a  number  of  reasons  why  this  should  be  so: 

First,  there  is  evidence  that  memories  are  associated  to  the  features  of  the  context  in  which 
they  were  learned.  The  probability  of  retrieving  the  memories  is  increased  when  the  context  of  recall 
matches  the  context  of  study  (Tulving,  1983;  Tulving  and  Thomson,  1973).  An  extreme  example  of 
this  was  shown  by  Ross  (1984)  who  found  that  secretaries  were  more  likely  to  remer  a  text-editor 
command  learned  in  the  context  of  a  recipe  if  they  were  currently  editing  another  n 

Second,  it  is  often  difficult  to  properly  encode  and  understand  information  p  nted  outside  of 
a  problem  context  and  so  its  applicability  might  not  be  recognized  in  a  problem  coni.  ..  For  instance, 
students  may  not  realize  that  a  top-level  variable  is  really  the  same  thing  as  a  function  argument  even 
though  they  are  obliquely  told  so.  As  another  example,  many  students  reading  the  side-angle-side 
postulate  may  not  know  what  included  angle  means  and  so  misapply  that  postulate. 

Third,  even  if  student  can  recall  tne  information  and  apply  it  correctly,  they  are  often  faced  with 
many  potentially  applicable  pieces  of  information  and  do  not  know  which  one  to  use.  We  have 
frequently  observed  students  painfully  trying  dozens  of  theorems  and  postulates  in  geometry  before 
finding  the  right  one.  The  basic  problem  is  that  knowledge  is  taught  in  the  abstract  and  the  student 
must  learn  the  goals  to  which  that  knowledge  is  applicable.  If  the  knowledge  is  presented  in  a 
problem-solving  context  its  goal-relevance  is  much  more  apparent. 

Private  human  tutors  characteristically  provide  information  in  the  problem-solving  context. 
Some,  but  not  all,  of  the  tutors  we  observed  gave  almost  non-stop  comment  as  students  tried  to  solve 
problems.  They  take  great  advantage  of  the  multi-modality  character  of  the  learning  situation- -with 
the  student  solving  the  problem  in  the  visual  modality  and  their  instruction  in  the  auditory  modality. 
Although  it  would  be  difficult  for  a  computer  tutor  to  be  as  interactive  as  these  human  tutors  and  take 
full  advantage  of  multi-modal  processing,  we  shall  see  that  it  is  possible  to  partially  achieve  this  by 
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Droviding  appropriate  instruction  tailored  to  the  students'  current  goal  context. 

Principle  3:  Provide  Immediate  Feedback  on  Errors 

Novices  make  errors  both  in  selecting  wrong  solution  paths  and  in  incorrectly  applying  the  rules 
of  the  domain.  Errors  are  an  inevitable  part  cf  learning,  but  the  cost  of  these  errors  to  the  learner  is 
often  higher  than  is  necessary.  They  can  severely  add  to  the  amount  of  time  required  for  learning. 
More  than  half  of  our  subjects’  problem  solving  sessions  were  actually  spent  exploring  wrong  paths 
or  recovering  from  erroneous  steps.  Relatively  little  is  learned  while  students  are  trying  to  get  out  of 
the  holes  they  have  dug  fc.  themselves. 

In  addition,  errors  often  confuse  the  picture  and  make  it  difficult  to  determine  which  steps  were 
right  or  wrong.  The  classic  example  of  this  is  the  student  who  finally  stumbles  on  the  correct  code  but 
does  not  understand  why  it  works.  Students  often  progress  in  this  trial  and  error  mode  with  respect  to 
LISP  evaluation:  they  don’t  know  when  an  element  will  be  treated  as  a  function,  a  variable,  or  a  literal 
but  play  around  with  parentheses  and  quotes  until  they  get  something  to  work.  It  is  particularly 
difficult  to  learn  Pom  errors  when  the  feedback  on  the  errors  comes  at  a  delay.  We  (Lewis  & 
Anderson,  submitted)  have  shown  that  subjects  learn  more  slowly  in  a  problem-solving  situation 
where  they  are  allowed  to  go  down  erroneous  paths  and  are  only  given  feedback  at  delay.  Also,  the 
importance  of  immediacy  of  feedback  has  been  well  documented  in  other  learning  situations 
(Bilodeau.  1959;  Skinner,  1958). 

Another  cost  of  errors  is  the  demoralization  of  the  student.  These  are  domains  in  which  errors 
can  be  very  frequent  and  frustrating.  We  believe  that  much  of  the  negative  attitudes  and  math 
phobias  derive  from  the  bitter  experiences  of  students  with  errors. 

The  advantages  of  a  private  tutor  are  clear  in  this  regard.  The  tutor  can  prevent  the  student 
from  wasting  inordinate  amounts  of  time  searching  wrong  paths.  The  tutor  can  both  provide 
immediate  feedback  when  errors  are  made  and  point  out  to  the  students  which  aspects  of  the 
problem-solution  are  correct  and  which  are  in  error. 
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Principle  4:  Minimize  Working  Memory  Load 

Solving  problems  can  require  holding  all  the  requisite  information  in  a  mental  working  memory. 
If  some  of  that  requisite  information  is  lost  there  will  be  errors.  It  surprised  us  to  find  out  in  our  LISP 
protocols  that  most  of  the  student  errors  appear  to  be  due  to  working  memory  failures.  A  frequent 
and  disastrous  type  of  error  is  "losing  a  level  of  complexity".  One  way  this  manifests  itself  is  that 
subjects  lose  track  of  one  level  of  parentheses.  Another  way  this  occurs  is  when  subjects  plan  to  use 
function  1  within  function2  within  function3,  but  forget  the  intermediate  function  and  write  function3 
directly  within  functionl . 

Working  memory  errors  are  frequent  in  geometry  but  less  frequent  than  in  LISP.  (Their  lesser 
frequency  is  because  the  diagram  and  the  proof  are  effective  records  of  inferences  made.)  One  place 
that  working  memory  errors  do  occur  is  when  students  work  backward  from  the  goal  and  create 
subgoals.  For  instance,  a  subject  will  reason  "I  can  prove  these  segments  congruent  if  I  can  show 
this  triangle  is  a  right  triangle.  I  can  do  that  if  I  can  show  its  other  two  angles  sum  to  90".  The  subject 
then  proves  the  two  angles  sum  to  90  but  forgets  how  this  fits  into  the  btgger  picture.  It  is  because  of 
the  high  working  memory  burden  that  students  are  so  bad  at  backward  inference  in  geometry. 

A  good  human  tutor  can  recognize  errors  of  working  memory  and  typically  provides  quioK 
correction  (McKendree,  Reiser,  and  Anderson,  1984).  Tutors  realize  that  there  is  little  profit  in 
allowing  the  student  to  continue  with  such  errors.  However,  human  tutors  really  have  no  easy  means 
at  their  disposal  for  reducing  the  working  memory  load.  This  is  one  of  the  ways  we  think  computer 
tutors  can  be  an  improvement  over  human  tutors  -one  can  externalize  much  of  working  memory  by 
rapid  updates  in  the  computer  screen.  This  involves  keeping  partial  products  and  goal  structures 
available  in  windows. 

Principle  5:  Represent  the  Student  as  a  Production  Set 

All  of  our  work  on  skill  acquisition  has  modeled  students'  behavior  as  being  generated  by  a  set 
of  productions.  There  is  a  fair  amount  of  evidence  for  this  view  of  human  problem-solving  (e.g., 
Anderson,  1983;  Newell  &  Simon,  1972).  It  is  also  the  case  that  numerous  other  efforts  in  the  domain 
of  intelligent  tutoring  have  taken  to  representing  the  to-be-tutored  skill  as  a  production  set  (eg.  Brown 
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and  Van  Lehn,  1980;  O’Shea.  1979;  Sleeman,  1982). 

Productions  in  ACT  represent  the  knowledge  underlying  a  problem  solving  skill  as  a  set  of 

goal-oriented  rules.  Some  representative  examples  for  LISP  and  geometry  are: 

IF  the  goal  Is  to  Insert  a  element  into  a  list 

i HEN  plan  to  use  CONS  and  set  as  subgoals 

1.  To  code  the  element 

2.  To  code  the  list 

IF  the  goal  Is  to  code  a  function  that  calculates  a  relation  on  a 
list 

THEN  try  to  use  CDR-recurslon  and  set  as  subgoals 

1.  To  code  the  terminating  condition 

2.  To  code  the  recursive  condition 

IF  the  goal  Is  to  prove  AXYZ  =  AUVW 
and  XT  s  OV 
and  TZ  s  VII 

THEN  plan  to  use  side-angle-side  and  set  as  a  subgoal 
1.  To  prove  ZXYZ  s  ZIIVW 

Such  rules  not  only  enable  the  system  to  follow  stuaent  problem-solving  but  they  define  an 
appropriate  grain  size  for  instruction.  Basically,  the  tutoring  systems  monitor  whether  a  student  has 
each  rule  correctly  and  corrects  any  incorrect  or  missing  rules.  As  emphasized  by  Brown  and  Van 
Lehn,  student  misconceptions  or  bugs  can  be  organized  as  perturbations  of  such  rules. 

Human  tutors  try  to  intuit  an  appropriate  grain  size  of  rules  for  instruction  but  often  their 
intuitions  are  wrong.  This  is  one  place  where  a  system  based  on  careful  analysis  of  student  problem¬ 
solving  may  be  able  to  outperform  the  typical  human  tutor. 

Principle  6:  Adjust  the  Grain  Size  of  Instruction  with  Learning 

One  of  the  reasons  human  tutors  have  difficulty  with  the  grain  size  for  instructing  students  is 
that  the  grain  size  cnanges  as  experience  is  acquired  in  the  domain.  According  to  the  ACT  learning 
theory  this  change  is  produced  by  a  knowledge  compilation  process  that  collapses  sequence  of 
productions  into  larger  "macro"  production  rules.  Human  tutors,  being  highly  skilled  in  the  domain 
exemplify  a  large  grain  size  in  their  problem -solving  and  have  a  considerable  difficulty  intuiting  the 
appropriate  grain  size  for  the  student. 


An  effective  computer  tutor  will  have  to  adjust  the  grain  size  of  instruction  as  the  student 
progresses  through  the  material.  Using  a  theory  of  production  leai  ning,  it  will  have  to  predict  when 
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the  original  productions  become  compiled  into  macro  productions  so  that  it  can  change  the  grain  size 
of  instruction. 

Principle  7:  Enable  the  Student  to  Approach  the  Target  Skill  by 

Successive  Approximation 

Students  do  not  become  experts  in  geometry  or  LISP  programming  after  solving  their  first 
problem.  They  gradually  approximate  the  expert  behavior,  accumulating  separately  the  various 
pieces  (production  rules)  of  the  skill.  It  is  important  that  a  tutor  support  this  learning  by 
approximation.  It  is  very  hard  to  learn  in  a  tutorial  situation  that  requires  that  the  whole  solution  be 
correct.  The  tutor  must  accept  partially  correct  solutions  and  shape  the  student  on  those  aspects  of 
the  solution  that  are  weak. 

Generally,  it  is  better  to  have  the  early  approximations  occur  in  problem  contexts  that  are  as 
similar  to  the  final  problem  space  as  possible.  Skills  learned  in  one  problem  context  will  only  partialiy 
transfer  to  a  second  context.  Students  are  learning  features  from  early  problems  to  guide  their 
problem-solving  operators.  If  these  features  are  different  from  the  final  problem  space  the  problem¬ 
solving  operators  will  be  misguided.  For  instance,  early  problems  in  geometry  tend  to  involve 
algebraic  manipulations  of  measures.  Consequently,  the  student  learns  to  always  convert 
congruence  of  segments  and  angles  into  equality  of  the  measure  of  the  parts.  Later  problems  such  as 
those  involving  triangle  congruence  do  not  involve  converting  congruence  of  sides  and  angles  into 
equality  of  measures.  Students  frequently  carry  over  their  overgeneral  tendency  to  convert  and  get 
into  difficulties  because  of  this. 

It  is  often  extremely  difficult  for  students  starting  out  to  solve  the  kind  of  problems  that  they  will 
eventually  have  to  solve.  For  instance,  in  geometry  students  cannot  initially  generate  proofs.  To  deal 
with  this,  standaid  pedagogy  often  evolves  intermediate  tasks  such  as  giving  reasons  for  the  worked 
out  steps  of  a  proof.  The  problem  is  that  the  process  of  finding  reasons  for  the  steps  of  a  proof  is 
different  than  the  process  of  generating  that  proof.  As  another  example,  in  programming  students  are 
often  given  practice  evaluating  recursive  functions  as  preparation  for  writing  recursive  functions. 
Again,  these  are  separate  iasks.  Both  in  the  geometry  and  the  programming  case  we  have  shown  that 
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there  is  not  much  transfer  from  one  task  to  another. 

The  advantage  of  a  private  tutor  is  that  he/she  can  help  the  student  through  problems  which 
are  too  difficult  for  the  student  to  solve  entirely  alone.  Thus,  it  is  common  to  see  a  sequence  of 
problems  where  the  tutor  will  solve  most  of  the  first  problem  with  the  student  just  filling  in  a  few  of  the 
steps,  less  of  the  second,  etc.  until  the  student  is  solving  the  entire  problem. 

Principle  8:  General  Problem-Solving  Rules  over  Analogy 

There  are  two  basic  methods  that  we  have  observed  students  u?'ng  to  solve  problems  in  these 
domains.  One  is  to  use  analogies  to  earlier  problems  in  Lhe  text  or  problems  from  other  domains  to 
help  guide  the  problem  solving.  The  basic  strategy  is  to  try  to  map  the  structure  of  a  solution  of  one 
problem  to  another  problem.  Anderson  (1981  tech  report),  Anderson,  Farrell,  and  Sauers  (1984),  and 
Anderson,  Pirolli,  and  Farrell  (in  press)  discuss  specific  examples  from  our  protocols  on  geometry  and 
LISP. 

The  other  method  is  to  extract  general  problem-solving  operators  from  the  instruction  and 
apply  these  to  the  problem.  For  instance,  if  the  goal  is  to  prove  triangles  congruent,  one  can  apply 
postulates  about  triangle  congruence.  If  the  goal  is  to  create  a  list  structure  one  can  try  to  apply  a 
function  that  creates  list  structures.  The  problem  with  such  general  operators  is  that  in  many 
domains  the  search  space  of  the  combinations  of  these  operators  becomes  enormous.  This  is 
perhaps  why  only  little  additional  information  tends  to  be  introduced  with  each  new  section  of  a 
textbook.  The  student  can  restrict  search  to  these  new  potential  operations  (cf.  Van  Lehn,  1983). 

Another  difficulty  with  the  general  problem-solving  approach  is  that  it  is  often  difficult  to  encode 
the  needed  problem-solving  operators.  Often  the  instruction  does  not  contain  explicit  statement  of 
such  operators.  Rather  the  operators  have  to  inferred  from  the  instruction.  Even  on  those  occasions 
in  which  the  operators  are  directly  stated,  students  have  a  hard  time  understanding  them  because 
they  are  stated  so  abstractly.  Students  are  often  only  able  to  encode  the  operators  correctly  when 
they  see  them  applied  to  an  example  problem. 

Subjects  appear  to  prefer  analogy  as  a  method  of  solution  in  both  geometry  and  LISP.  The 
preference  is  not  overwhelming  in  geometry  and  there  are  many  episodes  of  problem  solution  by 
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general  problem  solving  operators,  in  contrast,  the  preference  is  overwhelming  in  novice  LISP 
programming.  In  almost  every  case  where  a  student  was  writing  a  first  instance  of  a  particular  type  of 
LISP  function,  the  student  relied  on  analogy  to  example  LISP  tunctions. 

Private  human  tutors  differ  as  to  whether  they  tend  to  guide  the  student  to  solution  by  analogy 
or  by  general  problem-solving  operators.  We  claim  that  solution  with  general  operators  would  lead  to 
the  best  long-term  gains.  This  is  because  the  student  often  successfully  generates  a  solution  by 
analogy  but  does  not  understand  why  the  solution  works.  We  have  seen  students  work  their  way 
through  problems  by  analogy  and  not  learn  anything  of  permanent  value.  What  they  often  learn  is 
how  to  do  analogies.  Take  away  the  problems  from  which  to  analogize  and  they  are  unable  to  solve 
problems.  Halasz  and  Moran  (1982)  have  also  commented  on  the  negative  consequences  of  problem 
solving  by  analogy.  They  point  out  that  students  are  prone  to  incorrect  inferences  in  using  the 
analogy.  An  analogy  'S  frequently  used  to  replace  a  deep  understanding  of  the  problem  domain. 
Learning  is  going  to  be  limited  if  students  do  not  really  understand  the  domain. 

Given  the  importance  of  analogy  to  current  thinking  to  cognitive  science,  the  issue  of  analogy 
versus  general  problem-solving  needs  further  research.  However,  we  have  structured  our  tutors  to 
get  subjects  to  solve  the  problems  with  general  problem-solving  operators.  We  prevent  the  student 
from  having  access  to  previous  solution.  Rather  when  students  need  help  we  provide  them  with 
general  descriptions  of  the  relevant  problem-solving  operates. 

In  saying  that  we  discourage  problem  solving  by  analogy  we  are  not  saying  that  examples  are 
not  important  to  learning.  As  discussed  early  it  is  much  easier  to  understand  the  problem-solving 
operators  when  they  are  presented  in  the  context  of  an  example.  However,  we  try  to  keep  just  one 
example  in  front  of  the  student,  the  cur  rent  problem  being  worked  on,  and  not  give  the  subject  access 
to  prior  worked-out  examples. 

Issues  of  Human  Engineering 

We  have  tried  to  achieve  the  principles  enumerated  above  in  our  development  of  computer 
tutors.  However,  many  of  the  problems  that  we  face  in  creating  actual  computer  tutors  were  not  with 
achieving  these  principles  but  were  at  a  level  which  might  best  be  called  human-engineering.  This 
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refers  to  issues  of  designing  the  interface  and  the  natural  language  dialogue  so  that  information 
exchanges  occur  that  satisfy  our  cognitive  principles  Our  human  engineering  efforts  have  been 
somewhat  guided  by  what  we  know  from  cognitive  psychology,  somewhat  guided  by  results  in  the 
literature  on  the  human-computer  interface,  but  largely  the  result  of  trial  and  error  exploration.  This 
human-engineering  aspect  is  far  from  trivial.  Before  we  are  going  to  have  a  good  theory  of  intelligent 
tutors,  we  will  need  a  good  theory  of  their  human  engineering. 

In  the  remaining  two  sections  of  this  paper  we  will  describe  our  geometry  tutor  and  our  LISP 
tutor.  We  will  try  to  show  how  these  tutors  approximate  the  design  criteria  we  have  set  forth.  We  will 
also  will  try  to  communicate  some  of  our  experience  with  the  human-engineering  problems. 

The  Geometry  Tutor 

A  geometry  proof  problem  as  stated  in  a  high-school  geometry  text  (see  Fig.  1)  consists  of 
three  ingredients- -a  diagram,  a  set  of  givens,  and  a  to-be- proven  statement.  Despite  frequent  claims 
to  the  contrary,  the  diagram  frequently  plays  a  critical  role  in  the  logical  proof.  Typically  the  diagram 
of  the  only  source  of  critical  information  about  what  points  are  collinear  and  which  points  are 
between  which  others.  This  information  often  is  not  provided  in  the  givens.  Most  other  information 
which  can  be  read  off  the  diagram,  such  as  relative  measure,  is  not  to  be  taken  as  true  in  general.  It  is 
a  rare  high-school  proof  problem  that  involves  constructions  or  creating  new  entities  not  in  the 
diagram.  Indeed,  some  geometry  texthooks  have  a  policy  of  never  requiring  the  student  to  do  proofs 
by  constructions  although  all  conventional  texts  must  use  proof  by  construction  in  establishing 
theorems  for  the  student.  Therefore,  to  a  good  approximation,  the  student’s  task  is  to  find  some 
chain  of  legal  inferences  from  the  stated  givens  and  the  givens  implicit  in  the  diagram  to  the 
conclusion. 

Insert  Figure  4  about  here 

The  Ideal  Model 

Consider  the  problem  in  Figure  4.  There  are  a  set  of  forward  and  backward  deductions  that  can 
be  made.  Forward  deductions  take  information  given  and  note  that  certain  conclusions  follow.  So, 
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we  can  infer  from  the  fact  that  the  M  is  the  midpoint  of  TO  that  AM  =  MB.  We  can  also  infer  from  the 
vertical  angle  configuration  that  ZAMF  =  ZBME.  Backwaid  inferences  involve  noting  that  a 
conclusion  could  be  proven  if  certain  other  statements  were  proven.  So,  we  could  prove  M  is  tho 
midpoint  of  EF  if  we  could  prove  EM  =  MF.  We  could  prove  the  latter  statement  if  we  could  prove  EM 
=  AM  and  MF  s  AM.  As  these  examples  illustrate,  sometimes  forward  and  backward  inferences  are 
part  of  the  solution  and  sometimes  they  are  not.  In  challenging  problems  the  student  cannot  always 
know  whether  an  inference  is  part  of  the  solution.  Rather,  the  student  can  make  heuristic  guesses  at 
which  inferences  are  likely. 

In  our  view,  the  'deal  student  extends  the  proof  backward  from  the  to-be-proved  statement  and 
forward  from  the  givens  until  there  is  a  complete  proof.  At  each  point  the  ideal  student  makes  the 
heuristically  best  inference  where  this  is  defined  as  the  inference  most  probably  part  of  the  final  proof. 
"Most  probable"  depends  on  some  induction  over  the  space  of  high-school  problems.  Currently,  we 
have  no  formal  definition  of  the  probability  that  an  inference  will  part  of  a  proof,  but  our  intuitions  are 
usually  quite  defensible.  So,  we  create  the  ideal  student  model  as  a  set  of  rules  that  seem  heuristic. 
For  instance,  one  rule  is  that  when  vertical  angles  are  parts  ot  to-be-proven  congruent  triangles,  infer 
that  they  are  congruent  but  not  otherwise.  Basically,  the  rules  all  take  the  form  of  "apply  a  particular 
rule  of  inference  when  such  and  such  conditions  prevail".  These  conditions  refer  to  properties  of  the 
diagram,  givens,  and  established  inferences ,  and  goals  set  in  backward  inference.  These  rules  can 
be  represented  as  production  rules;  for  example: 

IF  AXYZ  and  AUVW  are  to-be-proven  congruent  triangles 

and  XYW  and  ZYO  are  intersecting  lines 
Then  infer  ZXYZ  =  ZUYW  because  of  vertical  angles. 

The  Proof  Graph 

As  noted  earlier,  standard  instruction  does  a  very  poor  job  of  communicating  the  goal  structure 
to  the  student.  Therefore,  our  first  human-engineering  problem  was  to  find  a  way  of  communicating 
this  information  to  the  student.  We  decided  to  use  a  graphical  formalism  in  which  the  to-be-proven 
statement  was  at  the  top  and  the  givens  were  at  the  bottom.  A  proof  is  created  as  a  logical  network 
connecting  the  givens  to  the  to-be-proven  statement.  The  basic  unit  of  this  network  is  a  structure 
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connecting  one  or  more  givens  to  a  conclusion  through  a  rule  of  inference.  Figure  5  shows  four 
states  of  the  proof  network  from  beginning  to  end  for  the  problem  in  Figure  4.  The  network  can  be 
grown  from  the  bottom  by  the  forward  inference  or  from  the  top  by  backward  inference.  As  Figure  5 
illustrates,  it  is  certainly  possible  to  generate  inferences  off  the  correct  path.  We  have  testimonials 
from  two  pilot  students  that  they  better  understood  both  the  structure  of  the  proof  and  nature  of  our 
proof  generation  because  of  the  graphical  formalisms  of  the  proof  structure. 

Insert  Figure  5  about  here 

Interacting  with  the  System 

The  basic  cycle  of  interaction  with  the  system  is  as  follows:  The  student  points  to  one  or  more 
statements  on  the  screen  from  which  he  or  she  wants  to  draw  an  inference.  If  at  least  one  legal 
inference  can  apply  to  these  the  statements  will  blink  and  the  system  asks  the  student  for  the  rule  of 
inference  to  apply.  The  student  types  in  the  rule  of  inference.  If  it  is  applicable  to  these  statements 
the  system  will  prompt  the  student  for  the  conclusion  that  follows.  The  student  types  in  the 
conclusion  or  points  to  it  if  it  is  on  the  screen.  If  correct  the  student  can  now  initiate  another  cycle  of 
interaction  with  the  system.  This  continue  until  a  complete  proof  has  been  generated. 

The  human-engineenng  of  the  system  has  involved  a  lot  of  trial  and  error  to  decide  issues  such 
as  how  to  position  the  graph,  what  aboreviations  to  use,  when  to  correct  misspellings,  how  to  let  the 
student  point,  and  how  to  relate  the  proof  structures  to  the  diagram.  We  have  also  found  it  useful  to 
have  the  student  spend  an  hour  with  a  warm-up  system  that  uses  the  same  graphical  conventions  but 
with  the  familiar  domain  of  arithmetic.  The  student  can  learn  to  use  the  system  much  more  rapidly  if 
this  learning  is  decoupled  from  learning  to  do  proofs  in  geometry.  One  problem  with  a  complex 
screen  is  that  the  student  may  not  notice  when  new  information  is  added.  We  have  found  that  color 
and  motion  are  fairly  effective  ways  of  capturing  attention.  So  we  have  taken  to  changing  the  colors 
of  the  windows,  bringing  up  new  windows  in  new  color,  or  blinking  critical  information. 

For  the  accomplished  geometry  student,  this  system  is  a  convenient  and  efficient  vehicle  for 
constructing  proofs.  It  serves  as  an  external  memory  so  that  forgotten  information  can  be  quickly 
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recovered.  It  also  catches  slips  of  mind  quickly.  However,  more  is  needed  when  dealing  with  a 
novice.  The  most  basic  problem  for  a  student  is  not  knowing  how  to  proceed.  There  are  a  number  of 
ways  that  our  system  helps  students  during  problem  solving.  Most  directly  the  student  can  ask  the 
system  for  help.  Less  directly,  the  student  may  choose  to  make  inferences  off  nodes  from  which  no 
inferences  or  no  useful  inferences  follow.  In  either  case,  the  system  will  provide  the  student  with  a 
hint  in  the  form  of  suggesting  the  best  nodes  from  which  to  infer. 

The  student  may  be  uncertain  about  what  inference  rule  to  apply  or  how  to  apply  the  inference 
rule.  He  can  manifest  this  again  by  asking  for  help  or  by  inappropriately  applying  a  rule.  In  this  case 
pop-up  windows  can  be  brought  up  to  display  which  rules  of  inference  are  currently  applicable  and  to 
display  a  aefinition  of  each  rule  of  inference.  When  the  student  displays  a  known  bug,  such  as 
applying  the  side-angle-side  postulate  when  the  angles  are  not  included  by  the  sides,  a  pop-up 
window  will  appear  explaining  why  the  student's  choice  is  not  correct. 

Students  get  in  ruts  in  which  they  try  to  make  an  inference  from  an  inappropriate  set  of 
statements  over  and  over  again  or  apply  the  same  rule  over  and  over  again  In  such  cases,  the  system 
will  interrupt  and  display  what  it  regards  as  the  next  best  inference  step  and  interrogate  the  student  to 
make  sure  that  the  step  is  understood. 

An  interesting  property  of  the  graphics  screen  is  that  the  student  has  no  access  to  solutions  of 
previous  problems.  Therefore,  it  is  very  difficult  to  do  any  problem  solution  by  analogy.  When  the 
student  gets  instruction,  the-  instruction  is  about  the  inference  rules  in  the  form  of  general  problem¬ 
solving  operators.  For  instance,  the  system  will  give  the  following  statement  of  the  corresponding 
parts  rule  when  it  is  evoked  in  backward  inference  to  prove  two  sides  congruent: 

IF  you  want  to  prove  the  conclusion  OV  s  XY 

THEN  try  to  prove  the  promise  AUVW  =  AXYZ 

along  with  a  pattern  diagram  to  help  the  student  instantiate  the  abstract  terms  in  this  statement. 

We  have  looked  intensively  at  three  students  working  with  the  system-  one  with  above  average 
ability,  one  of  average  ability,  and  the  other  of  below-average  ability  (as  defined  by  their  math  grades). 
The  below-average  student  came  to  us  for  remedial  purposes  having  failed  10th  grade  geometry.  The 
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other  two  students  were  eighth  graders  with  no  formal  geometry  experience.  We  can  only  report  that 
the  system  works- -that  all  students  learned  with  it  and  without  great  difficulty.  We  think  they  learned 
faster  than  with  traditional  instruction,  but  we  have  no  way  to  document  this  belief.  All  students  were 
able  to  do  problems  that  local  teachers  consider  too  difficult  to  assign  to  their  10th  grade  classes.  W*» 
are  currently  studying  additional  students  working  with  a  more  advanced  version  of  the  geometry 
system. 

The  LISP  Tutor 

Our  work  on  the  LISP  tutor  is  based  on  earlier  research  (Anderson,  Farrell,  Sauers,  1984; 
Anderson,  Pirolli,  and  Farrell,  in  press)  studying  the  acquisition  of  basic  LISP  programming  skills  by 
programming  novices.  Given  standard  classroom  instruction,  a  programming  novice  typically  takes 
over  50  hours  to  acquire  a  basic  facility  with  the  data  structures  and  functions  of  LISP.  At  the  end  of 
this  period  they  can  write  basic  recursive  and  iterative  functions.  In  this  time  tne  student  has  probably 
not  written  a  LISP  program  more  than  three  functions  deep  and  still  dees  not  know  how  to  use  LISP 
for  interesting  applications.  It  is  at  least  another  50  hours  before  the  student  has  the  facility  to  write 
modest  problem-solving  programs. 

We  believe  that  the  LISP  tutor  w'H  be  able  to  cover  the  same  material  in  under  20  hours  (we  are 
only  a  third  of  the  way  there  so  far).  After  this  point  the  tutor  would  step  back  and  become  an 
"intelligent  editor"  which  could  help  the  student  create  programs  and  catch  obvious  slips,  but  would 
no  longer  instruct.  Besides  the  desire  to  have  a  manageable  project,  we  do  not  feel  we  are  capable  of 
modeling  the  problem-solving  that  occurs  after  learning  the  basics  of  LISP. 

The  Ideal  Model 

Brooks  (1977)  analyzed  programming  into  the  activities  of  algorithm  design,  coding,  and 
debugging.  We  are  currently  focusing  on  algorithm  design  and  coding.  Our  past  research  on  LISP 
involved  creating  simulations  of  both  errorful  and  ideal  students  in  the  algorithm  design  and  coding 
phases.  Brooks  characterized  programming  as  first  designing  an  algorithm  and  then  converting  it 
into  code.  However,  the  break  is  seldom  so  clean.  The  student  alternates  between  algorithm  design 
and  coding,  sometimes  omitting  the  algorithm  design  altogether  and  going  directly  from  problem 
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statement  to  code.  Often  novices  and  experts  differ  as  to  whether  there  is  a  distinct  algorithm  design 
stage.  One  example  we  have  studied  involves  writing  LISP  code  to  take  a  list  and  return  that  list  with 
the  last  element  removed.  A  number  of  experts  generated  (REVERSE  (CDR  (REVERSE  LIST))) 
immediately  upon  hearing  the  statement  whereas  some  novices  went  through  a  10  minute  phase  of 
means-ends  analysis  to  come  up  with  the  algorithm. 

The  ideal  model  for  code  generation,  both  for  experts  and  novices,  involves  a  top  down 
generation  of  the  code.  Figure  6  illustrates  the  goal  structure  underlying  the  generation  of  the  code 
for  the  function  POWERSET  which  takes  a  list  and  returns  the  list  of  all  sublisis.  This  is  recognised  as 
involving  recursion  on  the  list  and  subgoals  are  set  to  code  the  terminating  step  and  the  recursive 
step  of  the  recursion  Both  of  these  steps  are  broken  down  into  algorithm  design  plus  code 
generation.  Writing  the  code  for  the  recursive  step  involves  writing  a  "helping  function"  ADDTO  that 
will  add  an  element  to  each  list  in  a  list  of  lists.  Novices  and  experts  differ  in  whether  they  will  try  to 
write  the  function  ADDTO  before  completing  the  code  for  POWERSET.  Novices  tend  to  be  uncertain 
exactly  what  ADDTO  will  do  and  whether  they  can  actually  write  such  a  function.  Therefore,  they  will 
interrupt  their  writing  of  POWERSET  to  code  ADDTO.  Experts  are  confident  that  ADDTO  can  be 
coded  and  postpone  i he  writing  of  ADCTO  until  completing  POWERSET. 

Insert  Figure  6  about  here 

The  actual  code  generated  along  with  the  goal  structure  in  Figure  6  is: 

(defun  powerset  (list) 

(cond  ((null  list)  (list  nil)) 

(t  (append  (powerset  (cdr  list)) 

(addto  (car  list)  (powerset  (cdr  list)))) 

))) 

There  are  a  number  of  features  to  note  about  Figure  6.  First,  code  is  generated  top-down. 
Second,  there  is  alteration  between  algorithm  design  and  code  generation.  Third,  the  encoding  of  the 
embedded  function,  ADDTO,  is  postponed  until  the  top  function  is  encoded. 

It  is  difficult  to  develop  ideal  models  for  the  algorithm  design  aspect  of  the  problem-solving. 
Students  can  potentially  bring  any  of  their  past  experiences  and  prior  knowledge  to  bear  in  designing 
an  algorithm.  With  each  problem  we  have  to  specially  provide  the  ideal  model  with  potentially 
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relevant  prior  knowledge.  So,  for  instance,  to  model  the  creation  of  a  graph  search  function  we  had 
to  provide  the  system  with  knowledge  of  the  fact  that  paths  in  a  graph  can  loop  (Anderson,  Farrell, 
and  Sauers,  1982). 

Interacting  with  the  LISP  tutor 

Figure  7  illustrates  a  typical  state  of  the  screen  in  interacting  with  the  tutor.  We  used  the 

hierarchical  structure  of  LISP  to  support  representation  of  the  top-down  structure  of  the  programming 

activity.  At  any  point  in  time  the  partial  code  is  displayed  before  the  student  in  one  window  with 

special  markers  indicating  the  structure  requiring  top-down  expansion:  eg 

(defun  rotator  (lis)  (append  (last  <1>)  <2>)) 

where  the  symbols  <1>  and  <2>  denote  the  points  for  top-down  expansion.  The  student  selects  a 
symbol  for  expansion  and  types  code  to  replace  this  symbol.  So,  at  this  level  the  system  is  just  a 
structured  editor  for  creating  the  code. 

Insert  Figure  7  about  here 

Sometimes,  the  hierarchical  structure  of  the  LISP  code  does  not  correspond  to  the  goal 
structure.  A  good  example  of  this  was  the  intersection  function  discussed  earlier  where  the  COND 
structure  had  three  clauses  but  the  goal  structure  for  CDR-recursion  just  involved  one  goal  for  doing 
the  terminating  case  and  one  goal  for  doing  the  recursive  step.  In  such  cases  we  have  the  subject 
generate  the  code  according  to  the  underlying  goal  structure  rather  than  the  syntax  of  the  LISP  code. 

Sometimes,  the  student  has  to  branch  into  algorithm  design  where  the  problem-solving  will  net 
correspond  to  the  hierarchical  structure  of  the  code.  In  this  case  a  window  appears  to  support  the 
development  of  the  algorithm.  Figure  8  illustrates  a  state  of  the  tutor  when  the  the  algorithm  window 
is  being  used  to  instruct  the  student.  The  final  product  of  the  algorithm  designing  will  be  a  specific 
window  of  code  that  must  be  mapped  into  the  abstract  code  of  the  function.  So,  for  instance,  if  the 
student  needs  to  design  an  algorithm  for  returning  a  list  of  all  but  the  last  of  the  LIS,  we  will  have  him 
work  out  a  solution  for  a  concrete  instance  of  LIS,  say  LIS  =  (A  B  C  D).  The  final  product  in  this  case 
would  be: 
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(REVERSE  (CDR  (REVERSE  '(A  B  C  D))) 
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which  can  be  mapped  into  the  abstract  function  to  produce: 

(REVERSE  (CDR  (REVERSE  LIS))) 

Again,  when  doing  the  recursive  step  *or  POWERSET  (see  Figure  6)  with  the  specific  list  (A  B  C),  the 
final  result  in  the  algorithm  window  would  be: 

(POWERSET  '(A  B  C))  = 

(APPEND  '(()  (C)  (B)  (B  C))  '((A)  (A  C)  (A  B)  (A  B  C))) 

and 

(POWERSET  ’(B  C))  =  (()  (C)  (B)  (B  C)) 

We  feel  that  the  code  window  and  the  algorithm  window  do  a  good  job  of  communicating  the 
hierarchal  structure  of  the  programming  activity  as  well  as  the  separate  status  of  algorithm  design. 

Insert  Figure  8  about  here 

There  is  a  third  window  in  which  the  student  communicates  to  the  tutor  his  choices  about  code 
and  algorithm.  Processing  the  code  is  relatively  straightforward  but  processing  the  algorithm  is  much 
more  difficult.  The  most  obvious  method  is  to  have  the  student  type  ir.  an  English  description  of  his 
algorithm  and  for  our  tutor  to  try  to  understand  that.  The  program  has  only  the  task  of  categorizing 
ihe  student's  description  into  one  of  the  algorithm  categories  it  is  prepared  to  process.  Even  with  this 
considerable  constraint,  we  have  only  had  modest  success  at  language  comprehension.  One  of  the 
frequent  student  complaints  about  the  tutor  is  its  inability  to  understand  algorithm  descriptions.  In 
part  this  is  due  to  the  tutor’s  limited  natural  language  understanding  and  in  part  this  is  because 
students  often  only  have  vague  ideas  of  algorithm  choices.  Our  solution  to  both  difficulties  has  been 
to  implement  a  menu  system. 

Our  menus  are  generated  dynamically  from  the  instantiations  of  productions  in  the  ideal  model. 
The  menus  contain  descriptions  of  all  the  algorithmic  variations,  correct  and  buggy,  that  we  are 
prepared  to  process.  Menu  selection  is  technically  simoler  than  language  comprehension.  It  is  an 
issue  for  further  research  which  is  more  effective.  We  were  surprised  by  how  well  students  appear  to 
adapt  to  menu  selection.  This  may  be  simply  that  it  is  much  easier  to  pick  a  menu  entry  than  to 
generate  a  description. 

As  in  the  geometry  system,  there  are  general  help  facilities  so  that  the  student  can  bring  up 
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information.  Again,  the  information  is  given  in  terms  of  abstract  problem  solving  operators.  For 
example,  rather  than  showing  how  CONS  is  used  in  a  specific  program,  CONS  might  be  explained:  "If 
you  want  to  insert  an  element  into  a  list  use  the  function  CuNS"  with  an  example  to  instantiate  the 
explanation. 

Novices  are  prone  to  a  substantial  set  of  misconceptions  and  slips  in  writing  LISP  programs. 
One  of  the  strengths  of  the  system  is  that  we  have  created  recognizers  for  a  great  many  of  the 
possible  bugs  and  provided  for  appropriate  feedback  on  the  errors.  In  our  pilot  study,  we  have 
discovered  a  number  of  other  stereotypic  errors  which  we  have  also  ente>ed  into  the  system  with 
appropriate  feedback. 

Table  1  illustrates  a  dialogue  with  our  tutor,  linearized  rather  than  developed  as  a  sequence  of 
screen  images.  This  interaction  demonstrates  how  the  tutor  recognizes  misconceptions,  provides 
immediate  feedback,  and  prevents  students  from  extensively  exploring  incorrect  paths. 

Because  of  the  availability  o*  the  college  population,  we  have  been  able  to  evaluate  the  quality 
of  our  LISP  tutor  more  carefully  than  we  have  the  geometry  tutor.  We  have  a  fairly  robust  version  of 
the  tutor  that  is  able  to  take  the  student  through  basic  LISP  functions  and  data  structures,  evaluation, 
function  composition,  and  function  definition.  It  culminates  in  a  series  of  six  exercises  in  function 
detinition.  We  estimate  that  students  would  take  on  the  order  of  10  hours  in  a  standard  classroom 
situation  to  listen  to  the  lectures,  read  the  text,  and  do  the  exercises,  in  contrast  to  that  situation, 
students  with  our  tutor  took  only  an  average  of  1  hr,  37  min. 

We  took  the  written  material  associated  with  the  computer  tutor  and  had  human  tutors  work 
personally  with  students.  These  human  tutors  were  undergraduates  and  graduates  who  knew  LISP 
well  and  who  had  previous  experience  tutoring.  In  this  situation,  they  took  1  hr,  21  r.iin.  to  work 
through  the  material.  As  a  final  condition  we  created  a  minimal  instructional  set  that  contained 
basically  just  the  information  in  the  computer  tutor.  Students  were  given  this  material  and  asked  to 
work  through  the  same  problems  as  in  the  other  condition.  There  was  a  consultant  available  should 
they  get  stuck.  In  this  case  subjects  took  2  hrs  15  minutes  to  get  through  the  material.  So  clearly 
either  tutoring  situation  improved  upon  the  on-your-own  condition.  The  fact  that  the  on-your-own 
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condition  is  better  than  the  classroom  probably  reflects  the  fact  that  than  standard  textbooks  or 
lectures  waste  a  lot  of  time  on  irrelevancies. 

Afterwards,  subjects  in  all  three  conditions  were  administered  a  standardized  test  of  knowledge 
of  LISP.  As  a  frame  of  reference,  students  after  an  estimated  30.5  hours  of  work  in  JRA’s  LISP  class 
got  57%  correct.  Students  with  the  computer  tutor  got  66%  correct.  Students  with  the  human  tutor 
got  69%  correct.  Students  left  on  their  own  got  72%  correct.  There  is  effectively  no  difference  in  the 
performance  of  any  of  the  three  experimental  groups  of  subjects,  although  they  all  seem  to  be  doing 
better  than  the  classroom  subjects. 

So,  a  private  tutor,  computer  or  human,  is  a  considerable  improvement  over  not  having  a 
private  tutor.  Humans  are  at  least  somewhat  better  than  our  computer  tutor,  Dut  we  were  looking  at  a 
first  pass  version  of  the  computer  tutor.  This  tutoring  experiment  produced  a  set  of  data  about  how  to 
do  tutoring  better  for  example,  unexpected  bugs  and  misconceptions.  Wo  really  think  it  is  feasib'e 
to  expect  that  one  can  construe!  computer  tutors  that  are  better  than  the  human  tutors  typically  found 
in  an  university  environment. 

Final  Conclusions 

This  is  very  much  a  report  of  work  in  progress.  We  have  yet  to  get  our  tutors  for  geometry  or 
LISP  into  their  final  states.  We  have  yet  to  completely  formalize  our  general  methodology  for  creating 
tutors.  Our  evaluation  is  still  very  preliminary,  both  of  the  general  methodology  and  of  the  specific 
tutors.  However,  we  believe  that  the  results  are  sufficiently  encouraging  to  deserve  a  report  now.  The 
basic  result  is  that  it  does  seem  possible  to  c.  eate  computer  tutors  that  are  capable  of  making  a  major 
improvement  in  the  education  of  a  wide  variety  of  topics.  An  important  additional  observation  is  that 
the  computer  technology  to  deliver  these  tutors  is  rapidly  becoming  economically  feasible. 

There  are  iwo  important  qualifications  to  make  about  any  projections  about  the  feasibility  of 
creating  intelligent  tutors.  First,  design  of  these  tutors  depends  critically  on  cognitive  science 
expertise.  It  is  not  yet  possible  for  a  cognitively  uninformed  system  designer  to  create  an  effective 
tutor  nor  for  a  computationally  uninformed  educational  psychologist  to  create  such  a  system.  We 
have  tried  to  distill  our  research  and  design  experience  into  a  set  of  piinciples,  but  these  principles 
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are  incomplete:  they  do  not  specify  how  to  create  student  models,  and  they  do  not  specify  how  to 
engineer  the  human-computer  interface.  Therefore,  creating  a  tutor  for  a  new  domain  poses 
research  problems  that  are  still  state  of  the  art  for  cognitive  science. 

Second,  developing  even  crude  prototype  systems  is  labor  intensive  and  requires  rather 
specialized  laborers  at  that.  We  expect  that  we  will  spend  10  man-years  of  effort  before  we  have 
satisfactory  tutors  for  full  courses  in  geometry  or  LISP.  Even  after  that  it  will  require  considerable 
effort  before  these  would  be  reliable  and  fine-tuned  enough  to  be  offered  as  commercial  products. 
We  hope  to  develop  our  methodology  enough  so  that  we  can  ct  the  10  year  figure  at  least  in  half. 
However,  it  seems  unlikely  that  we  will  ever  avoid  the  need  for  an  intensive  study  of  the  domain  and 
the  behavior  of  student  problem  solvers  . 
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Figure  Captions 

Figure  7.  An  example  of  a  two  column  proof  in  high-school  geometry. 

Figure  2.  A  representation  of  the  logical  structure  of  inferential  support. 

Figure  3.  A  representation  of  the  order  of  inferences  made  in  discovering  the  proof  in  Figure  3. 

Figure  4.  A  relatively  advanced  proof  problem  for  high  school  geometry. 

Figure  5.  Four  states  of  the  tutor  screen  in  a  students  construction  of  a  graph  proof  for  the 
problem  in  Figure  5. 

Figure  6.  A  representation  of  the  hierarchical  goal  structure  underlying  generation  of  the  LISP 
function  to  calculate  POWERSET. 

Figure  7.  A  typical  state  of  the  tutor  screen  in  interacting  with  a  student 

Figure  8.  A  state  of  the  tutor  screen  during  the  tutoring  of  the  algorithm.  Note  the  middle 
window  with  a  representation  of  the  current  state  of  the  algorithm  as  apllied  to  a  concrete  example. 
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TABLE  1 

Dialogue  with  LISP  Tutor 

I  want  you  to  define  a  function  called  ROTATER.  It  takes  1  argument  called  arg.  In 
defining  this  function  we'll  use  (a  b  c  d)  as  a  typical  example  of  arg.  This  function  is 
supposed  to  rotate  the  argument  list.  For  our  sample  case,  the  result  would  be  (d  a  b 


c). 


What  is  the  LISP  code  to  define  ROTATER? 

Define  rotater:  (defun  rotater  (arg)  (cons  (last  arg)  (cdr  arg))) 


CODE 

* 

(defun  rotater  (arg)  (cons  <1>  <2>))  * 

♦ 

***********«  +  %:****«************. ***%**#*«***.*-•* 


Your  code  for  <  1  >.  (last  arg)  produces  (d),  which  is  a  list  and  you  need  an  atom. 

What  LISP  code  are  you  going  to  write  to  get  the  last  element  of  the  argument  list? 
Your  answer  will  go  in  place  of  <1>  in  the  code.  You  are  trying  to  come  up  with  an 
atom,  like  d  in  our  example. 

Code  for  <1>:  d 

Your  code  for  <  I  >.  d,  is  a  good  example  for  the  last  element  of  the  argument  list,  but  it 
is  not  the  general  code.  We  need  LISP  code  that  will  evaluate  to  d  in  our  example 
case.  <1>  is  the  last  item  of  the  argument  list.  CAR  gets  the  first  element  of  a  list,  but 
since  this  item  is  the  last  element,  we  need  to  REVERSE  the  list  first  and  then  take  the 
CAR.  Taking  the  GAR  of  the  REVERSE  of  (a  b  c  d)  gives  us  d. 

**.»«4»*****«***************#***********1<******** 

*  CODE  * 

*  * 

*  (defun  rotater  (arg)  '* 

*  (cons  (car  (reverse  <3>))  <2>))  * 


What  are  you  going  to  write  to  go  in  place  of  <3>  in  the  code?  <3>  is  a  list.  A  typical 
example  of  the  result  is  (a  b  c  d). 
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Code  for  <3>:  arg 

Good.  This  makes  your  code  into  the  following: 

•  *****«****«***«********««,»*«********«**** 

•  CODE 

• 

•  (defun  rotater  (arg) 

*  (cons  (car  (reverse  arg))  <2>)) 


Your  previous  code  for  <2>,  (cdr  arg),  evaluates  to  (b  c  d),  while  what  you  want  is  (a  b 
c).  Remember  that  CDR  only  takes  an  element  off  of  the  front  of  a  list. 

What  LISP  code  are  you  going  to  write  to  get  all  but  the  last  of  the  argument  list?  Your 
answer  will  go  in  place  of  <2>  in  the  code.  You  are  trying  to  come  up  with  a  list,  like  (a 
b  c)  in  our  example. 

Code  for  <2>:  I  don't  know 

That  is  OK.  I’ll  help  you  out.  The  following  screen  will  display  a  menu  of  choices.  You 
have  to  decide  which  action  you  want  to  take  and  then  press  the  corresponding 
number.  One  option  on  every  menu  is  to  get  help  about  LISP  in  general. 

Remember  that  you  are  trying  to  get  all  but  the  last  of  the  argument  list.  You  are 

replacing  <2>  in  the  code.  The  result  is  a  list,  like  (a  b  c)  in  this  case. 

PRESS:  IF  YOU  WANT  TO: 

1  transform  (a  b  c  d)  into  what  we  want 

2  solve  for  separate  pieces  of  (a  b  c) 

3  have  the  tutor  choose 

4  get  information  about  LISP 

Menu  choice:  3 

OK.  I'll  tell  you  the  right  answer. 

We  ll  try  and  work  with  something  we  know  how  to  code  in  order  to  produce  the  code 
we  want.  We  ll  use  (a  b  c)  as  our  illustrative  example  for  the  tail  of  the  result  of  rotating 
the  argument  list  and  we  ll  use  (a  b  c  d)  as  our  example  for  the  argument  list. 


PLAN 


TRANSFORM:  (abed)  INTO  (a  b  c)  * 

* 


What  can  we  do  to  change  (abed)  into  (a  b  c)? 

PRESS:  IF  YOU  WANT  TO: 

1  remove  d  from  (abed) 

2  get  each  element  and  gather  them  into  a  list 

3  have  the  tutor  choose 

4  get  information  about  LISP 

Menu  choice:  2 

No.  That  is  not  general  enough.  If  you  tried  your  function  with  a  list  with  more  than  3 
elements,  it  would  not  work. 

Menu  choice:  t 
Sounds  good. 


**+********«*********«******«***«*****«***#**** 

*  PLAN  * 

*  * 

*  RCMOVE  d  FROM  (abed)  TO  GF.T  (a  b  c)  * 

*  • 
***«*****•»*«. ********************  ***************41 


What  function  will  remove  just  1  element  from  a  list  and  return  the  result? 

Function  name:  last 

If  you  apply  LAST  to  (a  b  c  d)  you  will  get  (d)  whereas  what  you  want  is  (a  b  c).  You 
want  to  remove  the  last  element  of  the  argument  list  not  extract  it. 

Let's  try  again. 

Function  name:  edr 

Good!  Your  answer  so  far  is  correct,  but  I  want  to  point  out  one  thing.  Remember  that 
in  this  case,  d  is  the  last  element  of  the  argument  list.  So,  in  order  to  apply  CDR,  you 
have  to  get  d  in  the  first  position  of  the  list.  We  will  represent  this  list  you  have  to  get 


by  ( d  ... ) 

What  can  we  do  to  change  (abed)  into  (d  ...)? 
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PRESS: 

1 

2 

3 

4 


Menu  choice:  1 


IF  YOU  WANT  TO: 

get  d  to  the  front  of  the  list 
go  down  the  list  until  you  get  d 
have  the  tutor  choose 
get  information  about  LISP 


That  answer  is  correct. 


*  *  *  4 

PLAN 

*  *  * 

* 

TRANSFORM: 

(abed)  INTO  (d  . 

*  *  *  *  * 

* 

* 

*  *  * 

What  function  will  get  d  to  the  front  of  the  list? 

Function  name:  reverse 

That  is  a  good  answer.  But,  remember  that  you  are  trying  to  get  (a  b  c).  The  plan  so 
far  for-getting  the  argument  list  minus  the  last  element  is  the  following:  (cdr  (reverse 
'(a  b  c  d)))  What  does  this  evaluate  to  in  LISP? 

Result:  (c  b  a) 

Very  good. 

*********************************************** 

•  example  code 

*  * 

*  (cdr  (reverse  ’(a  b  c  cl ) ) )  -  (c  b  a)  * 

*  * 

******************  **********  ***  * *************** 


What  can  we  do  to  change  (c  b  a)  into  (a  b  c)? 

PRESS:  IF  YOU  WANT  TO: 

1  successively  get  all  elements  of  (c  b  a) 

2  combine  (c)  with  (a  b) 

3  flip  over  ( c  b  a ) 

4  have  the  tutor  choose 

5  get  information  about  LISP 

Menu  choice:  1 

No.  We  don't  have  any  way  of  repeatedly  getting  elements.  If  we  did  have  a  procedure 
to  take  off  and  replace  exactly  3  items,  it  wouldn't  work  on  other  size  lists. 

We  ll  give  you  another  shot. 


Menu  choice:  3 


That  is  a  correct  answer.  Very  good. 


PLAN 

TRANSFORM:  (c  b  a)  INTO  (a  b  c) 


What  function  will  flip  over  (c  b  a)? 

Function  name:  don’t  have  the  faintest 
You  can  get  (a  b  c)  by  flipping  over  (c  b  a)  with  the  function  REVERSE. 

Current  Example  Code  — >  (reverse  (cdr  (reverse  (a  b  c  d))))  =  ,Jbc) 

Now  that  you’ve  written  code  to  achieve  (a  b  c),  what  is  the  general  LISP  code  to 
replace  the  code  for  our  example?  Your  answer  will  go  in  place  of  <2>  in  the  code. 

Code  for<2>:  (reverse  (cdr  (reverse  ’(a  b  c  d)))) 

That  is  the  code  using  the  example.  I  want  the  GENERAL  LISP  code,  using  the 
variables  in  our  function  definition. 

Let's  attempt  it  again 

Code  for  <2>.  (reverse  (cdr  (reverse  arg))) 

! ! !  YOU  ARE  DONE  !  ! ! 

**#*******^i»ir**«******<i<'#«li*#|r*,**ir,*>t«  *******  .#***#* 

*  FINAL  CODE  * 

*  * 

*  (defun  rotater  (arg)  * 

*  (cons  (car  (reverse  arg))  * 

*  (reverse  * 

*  (cdr  * 

*  ( reverse  arg ) ) ) ) )  * 

*  ****************************  *****#*«******'**** 


GIVEN:  XY  =  XZ, _ZWMY  =  Z  TMZ 
M  midpoint  of  YZ 

PROVE:  YT  =  ZW 


STATEMENT 

M  is  midpoint  of  YZ 
YM  =_MZ 
XY  =  XZ 
Z.XYZ  =  ZXZY 

Z  WMY  =  Z.TMZ 
4WMY_£/iTMZ 
WY  a  TZ 
AWYZj^TZY 
YT  ^ZW 


REASON 

Given 

Definition  of  midpoint 
Given 

base  angles  of 
isoceles  triangles 
Given 

Angle-side-angle  (ASA) 
Corresponding  parts 
side-angle-side  (SAS) 
Corresponding  parts 
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What  code  are  you  going  to  write  for  <1>.  This  should 
be  the  argument  list.  It  would  be  the  list,  (a  b  c  d),  in  the 
example. 


(defun  rotater  (lis) 

(append  (last  <1>)  <2>) 

Example:  (rotater  ’(abed))  =  (d  a b c) 


You  are  designing  an  algorithm  for  <2> 

You  want  to  get  a  list  of  all  but  the  last  item  of  the 
argument  list.  How  are  you  going  to  do  this? 

1 .  Get  all  elements  and  gather  therri  into  a  list. 

2.  Remove  D  from  (A  B  C  D) 

3.  Let  the  tutor  choose 

4.  Get  information  about  LISP 

>»2 


Algorithm: 

INITIAL:  (A  B  C  D) 

GOAL:  (A  B  C  D)  •*>  (A  B  C) 


Code: 

(defun  rotater  (lis) 

(append  (last  lis)  <2>) 


Example:  (rotater  ’(abed))  =  (d  a b c) 
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3001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  hr.  Janes  Baker 
Aray  Research  Institute 
3001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  mi  ton  S.  Katz 
Training  Technical  Area 
U.S.  Aray  Research  Institute 
3001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Marshall  Narva 
US  Aray  Research  Institute  tor  the 
Behavioral  t  Social  Sciences 
3001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Harold  F.  O'Neil,  Jr. 

Director,  Training  Research  lad 
Aray  Research  Institute 
3001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Coaaander,  U.S.  Aray  Research  Institute 
far  the  Behavioral  t  Social  Sciences 
ATTN:  PERI-BR  (Dr.  Judith  Orasanu) 

3001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Joseph  Psotka,  Ph.D. 

ATTN:  PERI-1C 

Aray  Research  Institute 

3001  Eisenhower  Ave. 

Alexandria,  VA  22333 

1  Dr.  Robert  Saseor 
U.  S.  Aray  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
3001  Eisenhower  Avenue 
Alexandria,  VA  22333 


Air  Force 

1  U.S.  Air  Fcrce  Office  of  Scientific 
Research 

Life  Sciences  Directorate,  ML 
Bolling  Air  Force  Base 
dashington,  DC  20332 

1  Dr.  Earl  A.  Aliuisi 
HO,  AFHRL  (AFSC) 

Brooks  AFB,  T1  7B235 

l  Mr.  Rayeond  E.  Lhristal 
AFHRL /NOE 

Brooks  AFB,  TX  78233 

1  Bryan  Dal  lean 
AFHRL /LRT 

Lowry  AFB,  £0  80230 

1  Dr.  Alfred  R.  Fregly 
AFOSR/NL 

Bolling  AFB,  DC  20332 

1  Dr.  6enevieve  Haddad 
Prograa  Manager 
Life  Sciences  Directorate 
AFOSR 

Bolling  AFB,  DC  20332 

1  Dr.  T.  H.  Longridge 
AFHRL /OTE 

Nilhaas  AFB,  AZ  33224 

1  Dr.  John  Tangney 
AFOSR/NL 

Bolling  AFB,  DC  20332 

1  Dr.  Joseph  Tasatuke 
AFHRL/LRT 

Lowry  AFB,  CO  30230 
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Departsent  o f  Defense 

^Defense  TKhnical  Intonation  Centtr 
Caaeron  Station,  Bldg  5 
Alexandria,  VA  22314 
Attn:  TC 

1  liilitary  Assistant  for  Training  and 
Personnel  Technology 
Office  of  the  Under  Secretary  of  Defens 
for  Research  t  Engineering 
Rooe  JD129,  The  Pentagon 
Nashington,  DC  20301 

1  Raj  or  Jack  Thorpe 
DARPA 

1400  Hi 1  son  llvd. 

Arlington,  VA  22209 

1  Dr.  Robert  A.  Misher 
OUSDRE  (ELS) 

The  Pentagon,  Rooe  3D129 
Nashington,  DC  20301 


Civillao  Agencies 

1  Dr.  Patricia  A.  Butler 
NIE-BRN  Bldg,  Stop  I  7 
1200  19th  St.,  RN 
Nashington,  DC  2020B 

1  Eduard  Esty 

Departeent  of  Education,  QER1 
NS  40 

1200  19th  St.,  NN 
Nashington,  DC  20208 

1  Dr.  Arthur  Nelaed 
724  Broun 

U.  S.  Dept,  of  Education 
Nashington,  DC  20208 

1  Dr.  Andrew  R.  Nolnar 
Office  of  Scientific  and  Engineering 
Personnel  and  Education 
National  Science  Foundation 
Nashington,  DC  20350 

1  Dr.  Frank  Nithroe 
U.  S.  Office  of  Education 
400  Maryland  Ave.  SN 
Nashington,  DC  20202 

1  Dr.  Joseph  L.  Young,  Director 
Resory  t  Cognitive  Processes 
National  Science  Foundation 
Nashington,  DC  20530 
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Private  Sedan 

1  *.  dvr  on  Barr 
Oopartsent  of  Coaputsr  kiwr 
Stanford  University 
Stanford,  C4  4*305 

1  Dr.  John  Hack 
fait  University 
loi  lid,  Tain  Station 
Nm  Havtn,  CT  04320 

1  Dr.  John  S.  Broan 
XEROX  Palo  Alto  Research  Center 
3333  Coyotn  Road 
Palo  Alto,  CA  44304 

1  Dr.  61tnn  Bryan 
6208  Poo  Road 
Bethesda,  HD  20817 

1  Dr.  Brucn  Buchanan 
Dopartaont  of  Coaputnr  Scitncn 
Stanford  University 
Stanford,  CA  4C303 

1  Dr.  Jaina  CarDontll 
Carnegie-Hellon  Univtriity 
Departnent  of  Psychology 
Pittsburgh,  PA  13213 

1  Dr.  Pat  Carpenter 
Dnpartnnnt  of  Psychology 
Carnngin-Hnllon  Univnrsity 
Pittsburgh,  PA  13213 

1  Dr.  Nichtlint  Chi 
Liarning  R  4  0  Cent nr 
Univnrsity  of  Pittsburgh 
3434  O'Hara  Strnot 
Pictsburgh,  PA  13213 

1  Dr.  dilliaa  Clancoy 
Dopartaont  of  Coaputnr  Sclnncn 
Stanford  Univnrsity 
Stanford,  CA  44304 

1  Dr.  Hichaal  Cola 
Univnrsity  of  California 
at  San  Dingo 

laboratory  of  Coaparativn 
Hunan  Cognition  ■  D003A 
la  Jolla,  CA  42043 


Privati  Sac tor 

1  Dr.  Allan  H.  Collins 
Bolt  Bnrannk  4  Kansan ,  Inc. 

30  Houlton  Strnot 
Casbndgn,  HA  02138 

1  Dr.  lynn  A.  Cooper 
IRCC 

University  of  Pittsburgh 
3434  O’Hara  Strut 
Pittsburgh,  PA  13213 

1  Dr.  Thgpas  H.  Duffy 
Dopartaont  of  English 
Carnagii-Hnllon  Univnrsity 
School ny  Park 
Pittsburgh,  CA  13213 

1  Dr.  Andors  Ericsson 
Onpartsnnt  of  Psychology 
Univnrsity  of  Colorado 
Soul  dor,  CC  80304 

1  Dr.  Paul  Fnltovich 
Dnpartsnnt  of  Hndical  Education 
Southorn  Illinois  University 
School  of  Hidicmo 
P.Q.  Bo*  3924 
Springfioid,  11  62708 

1  Hr.  Naliact  Feurzeig 
Departrent  of  Educational  Technology 
Bolt  Bnrannk  4  Neunan 
10  Houlton  St. 

Caabridgo,  HA  02238 

1  Dr.  Dmtnr  Fletcher 
Univnrsity  of  Oregon 
Departrent  of  Caeputer  Science 
Eugnnn,  OR  47403 

1  Dr.  John  R,  Frederiksen 
Bolt  Bnrannk  4  Reason 
30  Houlton  Strest 
Caabridge,  HA  02138 

1  Dr.  Hichael  Senesereth 
Departaent  of  Cosputer  Science 
Stanford  University 
Stanford,  CA  44303 

1  Dr.  Dodre  Sentner 
Bolt  Bnrannk  4  Reanan 
10  Houlton  St. 

Casbndgn,  HA  02138 
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Private  Sector 


Private  Sector 


1  Or.  Don  Stntner 

Center  for  Huean  InforeatJsR  Pr«essing 
University  of  California,  lee  Die?# 

La  Jolla,  CA  92093 


1  Dr.  Scott  Kelso 
Haskins  Laboratories,  Inc 
270  Craen  Street 
Nee  Haven,  CT  06510 


1  Or.  Robert  Slaser 

Learning  Research  l  Oevelopeent  Center 
University  of  RittsOurgh 
29*9  O’Hara  Street 
PITTSBURGH,  PA  13260 

1  Or.  Josph  Soguen 
SRI  International 
333  Ravensuood  Avenue 
Henlo  Park,  CA  9*023 

1  Or.  Daniel  Sopher 
Faculty  of  Industrial  Engineering 
6  Hanageeent 
TECHNIQM 
Haifa  32000 
ISRAEL 

1  Or.  Bert  Sreen 
Johns  Hopkins  University 
Departeent  of  Psychology 
Charles  6  34tb  Street 
Baltmore,  HD  21218 

1  OR.  JAHE5  8.  SREENO 
LRDC 

UHIVERSm  OF  PinSBURSH 
3939  O'HARA  STREET 
PITTSBURGH,  PA  13213 

1  Or.  Barbara  Hayes-Roth 
Departeent  of  Coeputer  Science 
Stanford  University 
Stanford,  CA  93303 

1  Or.  Frederick  Hayes-Roth 
Teknoeledge 
323  University  Ave. 

Palo  Alto,  CA  94301 

1  Dr.  Earl  Hunt 
Dept,  of  Psychology 
University  of  Hashington 
Seattle,  HA  98103 

1  Dr.  Harcel  Just 
Departeent  of  Psychology 
Carnegie-Hellon  University 
Pittsburgh,  PA  13213 


1  Or.  David  Kieras 
Departeent  of  Psychology 
University  of  Arizona 
Tuscon,  A2  83721 

1  Dr.  Halter  Kintsch 
Departeent  of  Psychology 
University  of  Colorado 
Boulder,  CO  80302  ♦ 

1  Dr.  Stephen  Kosslyn 
1236  Hilliae  Janes  Hall 
33  Kirkland  St. 

Caehndge,  HA  02138 

1  Dr.  Pat  Langley 
The  Robotics  Institute 
Carnegie-Hellon  University 
Pittsburgh,  PA  13213 

1  Dr.  Jill  Larkin 
Departeent  of  Psychology 
Carnegie  Hellon  University 
Pittsburgh,  PA  13213 

1  Or.  Alan  Lesgold 
Learning  RID  Center 
University  of  Pittsburgh 
3939  O’Hara  Street 
Pittsburgh,  PA  13260 

1  Dr.  Jie  Levin 
University  of  California 
at  San  Diego 

Laboratory  fof  Coeparative 
Hunan  Cognition  -  D003A 
La  Jolla,  CA  92093 

1  Dr.  Hichael  Levine 
Oepartaent  of  Educational  Psychology 
210  Education  81 dg. 

University  of  Illinois 
Chaepaign,  IL  61801 

1  Dr.  Harcia  C.  Linn 
Laurence  Hall  of  Science 
University  of  California 
Berkeley,  CA  94720 
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Private  SKtar 

1  Dr.  Jay  HcClelland 
Dtp art Mfl t  of  Psycho l oqy 
HIT 

Caebridqe,  HA  02134 

1  Dr.  JaMf  R.  Hilltr 
Coeputer»Thouqht  Corporation 
1721  Best  Plano  Hlqhnay 
Plano,  TX  73073 

1  Or.  Hart  Hi  11  or 
Coaputir*Tliouqht  Corporation 
1721  Hoot  Plano  Parkuay 
Plano,  TX  73073 

1  Dr.  Too  Horan 
Xerox  PARC 

3333  Coyote  Hill  Road 
Palo  Alto,  CA  44304 

1  Dr.  Allen  Hunro 

Behavioral  Technoloqy  Laboratories 
1843  Elena  Av».,  Fourth  Floor 
Redondo  Beach,  CA  40277 

1  Or.  Donald  A  Noroan 
Coqnitivo  Science,  C-013 
Univ.  of  California,  San  Dioqo 
La  Jolla,  CA  42043 

1  Dr.  Jesto  Or  Unsky 
Institute  for  Defense  Analyses 
1801  H.  Beaureqard  St. 

Alexandria,  VA  22311 

1  Prof.  Seyoour  Papert 
20C-104 

Hassachusotts  Institute  of  Technoloqy 
Caebridqo,  HA  02134 

1  Dr.  Haney  Penninqton 
University  of  Chicaqa 
Graduate  School  of  Business 
1101  E.  38th  St. 

Chicaqo,  IL  60637 

1  OR.  PETER  POLSOH 
DEPT.  OF  PSYCH0106Y 
URIVERSITY  OF  COLORADO 
BOULDER,  CO  80304 


Private  Sector 

1  Dr.  Fred  Reif 
Physics  DepartMnt 
University  of  California 
Berkeley,  CA  44720 

1  Dr.  Lauren  Resnick 
LRDC 

University  of  Pittsburqh 
3434  O’Hara  Street 
Pittsburqh,  PA  1521 

1  Hary  S.  Riley 
Proqrae  in  Coqnitive  Science 
Center  for  Husan  Inforeation  Processinq 
University  of  California,  San  Dieqo 
La  Jolla,  CA  42043 

1  Dr.  Andre*  H.  Rose 
Aoencan  Institutes  for  Research 
1033  Thotas  Jefferson  St.  HR 
Hashinqton,  DC  20007 

1  Dr.  Ernst  Z.  Rothkopf 
Bell  Laboratories 
Hurray  Hill,  NJ  07974 

1  Dr.  Hi  1 1 i as  B.  Rouse 
Georqia  Institute  of  Technoloqy 
School  of  Industrial  t  Systees 
Enqineerinq 
Atlanta,  SA  30332 

1  Dr.  David  Rueelhart 
Center  for  Huean  Inforeation  Processing 
Univ.  of  California,  San  Dieqo 
La  Jolla,  CA  42093 

1  Dr.  Hichael  J.  Saaet 
Perceptronics,  Inc 
6271  Variel  Avenue 
Woodland  Hills,  DA  91364 

1  Dr.  Roqtr  Schank 
Yale  University 

Department  of  Coeputer  Science 

P.0.  Box  1158 

N»n  Haven,  CT  06320 

1  Dr.  Walter  Schneider 
Psychol oqy  Departaent 
603  E.  Daniel 
Chaepaiqn,  IL  61820 
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Private  Sector 

1  Or.  Alan  Schoenfeld 
Mathematics  and  Education 
The  University  of  Rochester 
Rochester,  NY  14627 

1  Mr.  Colin  Sheppard 
Applied  Psychology  Unit 
Adeiralty  Marine  Technology  Est. 
Teddington,  Middlesex 
United  Kingdoe 

1  Dr.  H.  Wallace  Sinaiko 
Prograa  Director 

Nancower  Research  and  Advisory  Services 
Suthsonian  Institution 
801  North  Pitt  Street 
Alexandria,  VA  22Z14 

1  Dr.  Edward  E.  Saith 
Bolt  Beranek  4  Newaan,  Inc. 

30  Moulton  Street 
Cartridge,  MA  02138 

1  Dr.  Eliott  Soloway 
Yale  University 

Departaent  of  Coaputer  Science 

P.0.  Box  2138 

Nea  Haven,  CT  06320 

1  Dr.  Kathryn  T.  Spoehr 
Psychology  Departaent 
Brown  University 
Providence,  RI  02912 

1  Dr.  Robert  Sternberg 
Dept,  of  Psychology 
Yale  University 
Box  11A,  Yale  Station 
New  Haven,  CT  06320 

1  Dr.  Albert  Stevens 
Bolt  leranek  4  Newaan,  Inc. 

10  Moulton  St. 

Caabndge,  MA  02238 

1  DR.  PATRICK  SUPPES 
INSTITUTE  FOR  MATHEMATICAL  STUDIES  IN 
THE  SOCIAL  SCIENCES 
STANFORD  UNIVERSITY 
STANFORD,  CA  94303 


Private  Sector 

1  Dr.  Kikuai  Tatsuoka 
Coaputer  Based  Education  Research  Lao 
232  Engineering  Research  Laboratory 
Urbana,  1L  61801 

1  Dr.  Maurice  Tatsuoka 
220  Education  Bldg 
1310  S.  Sixth  St. 

Chaapaign,  IL  61B20 

1  Dr.  Perry  d.  Thorndyke 
Perceptronics,  Inc. 

343  Hiddlefield  Road,  Suite  140 
Menlo  Park,  CA  94023 

1  Dr.  Douglas  Towns 
Umv.  of  So.  California 
Behaviors'  Technology  Labs 
1B45  S.  Elena  Ave. 

Redondc  Beach,  CA  90277 

1  Dr.  Kurt  Van  Lehn 
Xerox  PARC 

3333  Coyote  Hill  Road 
Palo  Alto,  CA  94304 

1  Dr.  Keith  T.  descour t 
Perceptronics,  Inc. 

345  Middlafield  Road,  Suite  140 
Menlo  Park,  CA  94025 

1  Dr.  Thoaas  dickens 
Departient  of  Psychology 
Franz  Hall 

University  of  California 
403  Hilgarde  Avenue 
Los  Angeles,  CA  90024 

1  Dr.  Mike  dilliais 
IntelliSenetics 
124  University  Avenue 
Palo  Alto,  CA  94301 


